NPU user guide

Copyright © Quectel Wireless Solutions Co., Ltd. 2026. All rights reserved.

What is NPU

NPU (Neural Processing Unit) is the dedicated AI inference hardware integrated in the RK3576, with a computing power of 6 TOPS (INT8). It executes the core computations of neural networks (convolutions, matrix multiplications) in hardware in parallel, 10~25 times faster than the CPU.

What Can the NPU Do

Application

Description

Object detection

Identify objects in the image (people, cars, cats/dogs…)

Image classification

Determine which category an image belongs to

Face recognition

Detection / comparison / feature extraction

OCR

Text recognition

Pose estimation

Identify human keypoints

On the M2 board, running AI models (such as YOLO object detection) uses the NPU for accelerated inference. This guide walks you through it step by step.

Onboard Resources (Out of the Box)

The M2 Debian system comes with a complete NPU toolchain:

Resource

Path

NPU driver

rknpu v0.9.8

Inference runtime

/usr/lib/librknnrt.so (v2.3.2)

Inference test tools

/usr/bin/rknn_benchmark, rknn_common_test

Prebuilt model

/usr/share/model/RK3576/mobilenet_v1.rknn

Scenario 1: Run an Image Classification Model (Quickest Start)

Step 1: Verify NPU is ready

# NPU driver version
cat /sys/kernel/debug/rknpu/version
# Expected: RKNPU driver: v0.9.8

Step 2: Run mobilenet classification inference

# Run one inference with the prebuilt mobilenet_v1 model + test image
rknn_common_test /usr/share/model/RK3576/mobilenet_v1.rknn /usr/share/model/dog_224x224.jpg 3

Step 3: View results

The output shows model info, inference time, and classification results. M2 measured mobilenet_v1 single-frame inference is about 2.1ms (470 FPS) — this is the effect of NPU acceleration; the CPU takes more than 10x longer for the same model.

Scenario 2: Run YOLO Object Detection (Most Common)

Full YOLO detection steps are in the [YOLO On-Device Deployment for Object Detection] document. Brief here:

# Run on board (in the model directory)
cd /usr/share/model/yolov5
rknn_yolov5_demo /usr/share/model/yolov5/RK3576/yolov5s-640-640.rknn \
  model/bus.jpg resize /tmp/bus_detected.jpg

Measured: 19ms per frame (about 50 FPS), detecting 4 persons + 1 bus.

Scenario 3: NPU Performance Test (Multiple Models)

# Onboard performance test tool
rknn_benchmark /usr/share/model/RK3576/mobilenet_v1.rknn

RK3576 NPU measured performance (rknn_benchmark):

Model

Input Size

Avg Time

FPS

Type

resnet18

224x224

3.50ms

285

Image classification

resnet152

224x224

19.35ms

51.7

Image classification

yolov5s

640x640

17.37ms

57.6

Object detection

yolo11s

640x640

33.24ms

30.1

Object detection

yolov5s at 57 FPS > 30 FPS (video frame rate) means the RK3576 can do real-time video object detection.

Scenario 4: Run Your Own Model (Advanced)

Convert your model (PyTorch/TensorFlow/ONNX) to RKNN format to run on the NPU.

Step 1: Install conversion tools (PC side)

cd <SDK>/external/rknn-toolkit2/rknn-toolkit2
pip3 install packages/x86_64/rknn_toolkit2-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pip3 install "onnx==1.14.1"

Step 2: Export ONNX model

import torch
model = torch.hub.load('pytorch/vision:v0.14.1', 'resnet18', pretrained=True)
model.eval()
torch.onnx.export(model, torch.Tensor(1,3,224,224), 'resnet18.onnx',
                  input_names=['input'], output_names=['output'], opset_version=12)

Step 3: Convert to RKNN

from rknn.api import RKNN
rknn = RKNN()
rknn.config(mean_values=[123.675,116.28,103.53], std_values=[58.395,58.395,58.395],
            target_platform='rk3576')   # ★ Must specify rk3576
rknn.load_onnx(model='resnet18.onnx')
rknn.build(do_quantization=True, dataset='dataset.txt')
rknn.export_rknn('resnet18.rknn')

Step 4: Push to board and run

adb push resnet18.rknn /data/local/tmp/
adb shell "rknn_benchmark /data/local/tmp/resnet18.rknn 2>&1 | grep -i avg"
# Expected: Avg Time 3.50ms, Avg FPS = 285.7

Reproducible verification package: rknn-pytorch-demo.zip in the SDK root directory (includes resnet18/resnet152/yolo11s conversion scripts + one-click test).

FAQ

Problem

Solution

failed to open model

The model must be converted for the rk3576 platform; check the path

rknn_benchmark not found

On SMARC (Buildroot), use rknn_common_test instead

onnx.mapping error during conversion

onnx version must be ≤ 1.14.1

Slow inference

Fix NPU frequency to max 950MHz (see below)